Interactive purchasing of products displayed in video

ABSTRACT

Methods and the systems for providing an interactive, program based product introduction experience are described. In one example, video content may be captured, enriched with product data, and provided on-demand to interactive user devices. Users viewing the content may be presented with supplemental information or resources allowing those users to obtain more information about objects appearing in the video, or purchase products that may be a sociopathic with those objects. The objects appearing in the video may be passively up hearing, rather than explicitly introduced or offered for sale as part of the original video content. However, the supplemental information allows such users to discover and/or purchase new products through interactivity with the video content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 17/012,226,filed Sep. 4, 2020, which claims priority the benefit of provisionalapplication Ser. No. 62/896,388, filed Sep. 5, 2019, which applicationsare incorporated herein by reference in their entirety.

The present disclosure is directed to methods and systems for generatingand providing on-demand interactive content via video and streamingdevices.

BACKGROUND

Website shopping provides static images and video of products that areoffered for sale. Home shopping networks provide potential customers whoview a television program the ability to call or use the Internet toview and order products that are discussed during the televisionprogram. These product offerings systems have drawbacks. In the case ofweb site shopping, the extent of user interactivity is limited becausethe display of the product relates to the product alone, rather thandisplaying the product in the context of its use in a room, orcommentary by a knowledgeable spokesperson. In the case of home shoppingnetworks, a knowledgeable spokesperson may discuss a product, but thatdiscussion is unnatural because it focuses specifically on sales of theproduct, rather than discussing the product in the context of othertopics. Still further, in the case of home shopping networks, a useroften is required to use a different mechanism to actually order adesired product, typically by phone or Internet. Accordingly,improvements in purchasing products that display the products in theiruseful environments are desirable.

SUMMARY

Generally, the present disclosure relates to methods and systems forenabling consumers to conduct interactive purchasing of products thatare offered for sale in moving video.

In a first aspect, a system for generating and providing on-demandinteractive content to an audience is disclosed. The system includes acontent server including a processor and a memory, the memory hosting acontent database, a product database, and an advertising database, thememory further storing instructions which, when executed, cause thecontent server to receive video content, the video content including ashow, such as a talk show, having a plurality of scenes, each of thescenes including a host, one or more guests, and a plurality of objects;ingest the video content into the content database; process the videocontent to automatically identify an object of the plurality of objectsas associated with a product having an associated product record in theproduct database; define one or more content enrichment actions to beperformed during playback of the content, the one or more contentenrichment actions including generating an overlay providing productdetails of the product based on information about the product in theproduct database, the overlay being displayed in association with thecontent during playback of a portion of the content during which theobject appears; and define one or more advertisements to be displayed inassociation with the content during playback of at least a portion ofthe content. The system also includes an application servercommunicatively connected to the content server and configured toexecute instructions which cause the application server to perform thefunctions of receiving a request from an interactive user device forplayback of the video content; in response to the request, obtainingcontent, product information, and advertisement information form thecontent server; and generating a content feed to the interactive userdevice, wherein the content feed allows interaction with the contentfeed during playback of the video content based on the one or morecontent enrichment actions.

In a second aspect, a method of creating interactive retail-focusedvideo content is disclosed. The method includes capturing video contentof one or more scenes, each of the scenes including a plurality ofobjects, which may consist of products offered for sale oradvertisements. The method may optionally include a cast, such as a hostor guests, that appear in the scenes. The method further includesingesting the video content into a content database. The method alsoincludes processing the video content to automatically identify anobject of the plurality of objects as associated with a product havingan associated product record in a product database. The method includesdefining one or more content enrichment actions to be performed duringplayback of the content, the one or more content enrichment actionsincluding generating an overlay providing product details of the productbased on information about the product in the product database, theoverlay being displayed in association with the content during playbackof a portion of the content during which the object appears. The methodmay optionally include defining an advertisement to be displayed inassociation with the content during playback of at least a portion ofthe content. The method may further optionally include, upon receipt ofa request for the video content, providing the video content, the one ormore content enrichment actions, and the advertisement for playback.

In a third aspect, a method of providing interactive retail-focusedvideo content for playback is disclosed. The method includes receiving,from an application executing on a consumer computing device, a requestfor interactive video content, the request identifying specified videocontent. The specified video content includes one or more scenesincluding a plurality of objects. The specified video content mayoptionally include one or more scenes with a host, and may furtheroptionally include one or more guests. The method includes retrievingthe specified video content as well as product data, advertising data,and one or more content enrichment actions to be performed duringplayback of the specified video content. The method further includesproviding interactive video content to the consumer computing device,the interactive video content comprising the specified video content andexecuting the one or more enrichment actions during playback of thecontent, the one or more enrichment actions including one or more ofdisplaying an option to purchase a product corresponding to an object ofthe plurality of objects while the object is displayed during playbackof the specified video content; displaying an advertisement associatedwith an object of the plurality of objects while the object is displayedduring playback of the specified video content; and displaying anadvertisement associated with a topic of conversation while the topic ofconversation is addressed during the talk show as an overlay on aportion of the specified video content.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, not is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrate a diagram of an environment that may be used togenerate content for an on-demand interactive content system.

FIG. 2 illustrates an example block diagram of a computing system usefulfor implementing the on-demand interactive content system.

FIG. 3 illustrates a system for generating and providing on-demandinteractive content to an audience, and in particular content related toobjects available for purchase, according to an example embodiment.

FIG. 4 illustrates a content management portion of a system forgenerating and providing on-demand interactive content to an audience.

FIG. 5 illustrates a diagram of the formation of video content that maybe used as part of a system for generating and providing on-demandinteractive content to an audience.

FIG. 6 illustrates a production management system for producingon-demand interactive content that can be shown to an audience.

FIG. 7 illustrates a first scene produced as part of on-demandinteractive content and depicts that scene being shown via aninteractive device.

FIG. 8 illustrates a second scene produced as part of on-demandinteractive content and depicts that scene being shown via aninteractive device together with overlay content.

FIG. 9 illustrates a third scene produced as part of on-demandinteractive content and depicts that scene being shown via aninteractive device together with overlay content

FIG. 10 illustrates a scene produced as part of on-demand interactivecontent and depicts that scene being shown via an interactive devicetogether with overlay content and the selection of a product offered forsale via the interactive content.

FIG. 11 illustrates one embodiment wherein the objects in theenvironment are recognized using photo capture technology.

FIG. 12 illustrates an example methodology for identification of objectswithin video data using photographs, as described in conjunction withFIG. 11 .

FIG. 13 illustrates another embodiment for identification of objectswithin video data using photograph using a grid layout, as compared tothe embodiment discussed in conjunction with FIG. 12 .

FIG. 14 shows a general flowchart of a method of receiving andprocessing program content for enrichment and playback, in accordancewith example aspects of the present disclosure.

FIG. 15 shows a general flowchart of a method of interacting withenriched video content in accordance with the methods and systemsdescribed herein.

DETAILED DESCRIPTION

Various embodiments will be described in detail with reference to thedrawings, wherein like reference numerals represent like parts andassemblies through the several views. Reference to various embodimentsdoes not limit the scope of the claims attached hereto. Additionally,any examples set forth in this specification are not intended to belimiting and merely set forth the many possible embodiments for theappended claims.

Whenever appropriate, terms used in the singular also will include theplural and vice versa. The use of “a” herein means “one or more” unlessstated otherwise or where the use of “one or more” is clearlyinappropriate. The use of “or” means “and/or” unless stated otherwise.The use of “comprise,” “comprises,” “comprising,” “include,” “includes,”and “including” are interchangeable and not intended to be limiting. Theterm “such as” also is not intended to be limiting. For example, theterm “including” shall mean “including, but not limited to.”

As briefly described above, embodiments of the present disclosure aredirected to methods and systems for generating and providing on-demandinteractive content to an audience. Generally, this includes creatingvideo data based on an environment that includes objects that areavailable for sale, performing content enrichment actions on that videodata, and serving video data with content enrichment data such asproduct data, advertising data, and audio data (“enhanced video data”)to an audience via interactive devices. The audience, who includeconsumers, may interact with the enhanced video data and obtaininformation related to the objects shown in the enhanced video data. Theobjects include products that are tagged with content enrichment data orother items that are identified in video data to be displayed withadditional product data. The audience may purchase the objects byselecting the object during the playback of the enhanced video data.

In example implementations, an environment is created such as a backyardscene. The backyard scene includes one or more objects. In thisimplementation, the objects are products that are available for sale. Ahost with an optional guest is recorded in the environment using one ormore video cameras. The video cameras record the scene, including theinteractions of the host and guest and the objects that are in theenvironment. The video is uploaded to a network to a content server andstored in a video database. A computer system is used to perform contentenrichment actions on the video data. One example of a contentenrichment action is to generate an overlay that provides informationrelated to the object, including information related to purchase theobject. The enhanced video content is served to an audience viainteractive user devices such as a mobile phone, tablet, laptop,personal computer, or television. The user views the enhanced video datavia the interactive user device and selects an object to purchase thatis depicted in the enhanced video data via an ecommerce server.

Objects may be identified manually or automatically through imagerecognition, beacon technology, grid layout, and other forms thatidentify an object. In an example using image recognition, frames arescanned using image recognition technology at selected intervals. If anobject appears in a scanned frame, the computer system accessesinformation about that object and displays product data related to thatobject on the screen. As the frames advance, the computer systemcontinues to scan the frames to understand where the product is in thescreen and to display the product data in a preferred location withinthe screen. Editors may use video editing software to identify apreferred location to place the product data. As an object exists thescreen, or gets to a defined border as provided by the editor, thecomputer system recognizes that based on the frame screening and stopsdisplaying product data related to that object.

One or more objects may be displayed at a single time within the screen.In one embodiment, two objects are displayed on a single screen at atime interval. In this embodiment, the product data is available to bedisplayed, but remains hidden, until user selection. The user viewingthe video has the option to select one object, for example via touch,and view product data related to that particular object. If a secondobject is within the same screen during that same time interval, theuser may select the second object and product data will appear regardingthat second object and the product data for the first object willdisappear. Settings may be adjusted to allow product data for multipleobjects to be shown simultaneously or automatically without userintervention. User selection may include the use of a touch screen,audible command, mouse, remote control, infrared, motion, or any othermeans to select the object. A user may also forward, reverse, or pausethe video to better interact with the objects displayed on the screen ata particular time.

Another embodiment relates to the use of a beacon or other wired orwireless transmitter that sends signals to the video camera when theobject appears in the lens of the video camera. A beacon is placed on anobject within a scene. A video camera fitted with a receiver to receivethe transmission from the beacon is used to record video data. As thevideo camera pans towards an object fitted with a beacon, the signal isembedded within the video for the period of time that particular objectis in the screen. A user may adjust settings to activate the beacon whenthe full object is in the screen or when a portion of the object is inthe screen. Product data can then be linked to the location within thevideo data. In this embodiment, a video camera moving across a scene mayhave multiple objects come into the screen of the video camera. A videocamera, fitted with a transmission receiver, will tag the video datawhen the beacon from the object is within view of the video camera andthus create a tag in the video data that can be updated with productdata, advertising data, audio data, and other content enrichment data.That enhanced video data can then be transmitted to users who useinteractive devices. The users may select an object as it appears on thescreen. Upon selection, content enrichment data may be displayedtogether with the object. The user may be select the object forpurchase. An ecommerce server will serve to complete the purchase.

Another embodiment relates to the use of a stage to create anenvironment where objects are displayed in a scene on stage. Forexample, a stage may include two chairs with a table such that a hostand guest may converse. Surrounding the cast is a living room set withlamps, tables, sofas, other furniture, and decorations. A grid patterncan be established to identify the location of the objects as shown inthe video data. The grid can be used to tag an object with product data.Users interacting with the video that included enhanced video data mayselect objects and the system will display product data related to thatobject. Users may select an object for purchase via the system using anecommerce server. Any combination of elements related to the foregoingmay be combined and used with the system.

Another embodiment relates to the use of electronic transmitting tagsthat are placed on the cast members. Electronic transmitting tags arealso placed on or near objects within the environment. As a member ofthe cast comes near an object that has an electronic transmitting tag,the computer system records the time stamp of the video. The video canthen be edited to update the video data with enriched data such asproduct data, advertising data, and audio data. The tags may also belinked to the video camera to automatically identify the objects withspecific frames in the video data.

Referring to FIG. 1 , an environment 2 to create video content for ashow that includes one or more objects is shown, according to an exampleembodiment. The environment 2 is a backyard scene that includes a house4, fence 6, trees 8, garden 10, patio 12 with a table and chairs 14 andstring lights 16. The example environment 2 also includes a walkway 18that traverses the backyard scene to a pool 20 that contains floatingorbs 22. The example environment also includes a set of chairs 24, afire pit 26, a misting umbrella set 28, and a table with food 30. Avideo camera 32 is set in the environment 2 to record the scene. Aphysical advertisement 34 is included in the environment 2. A cast 36,which includes a host and a guest in the example embodiment, is withinthe environment 2.

The environment 2 can be any scene and should not be limited to abackyard scene. Other example environments can include rooms in a homesuch as a sunroom that includes objects such as billowy curtains, whiteswinging chairs, Nantucket style tables and planked chairs, charmingminiature birdhouses, aromatic diffusers, turquoise pillows and matchingrugs. Lobsters, crab cakes, mint topped drinks, and sunflowers may beplaced on tables within the environment. Another example environmentincludes a living room in modern skyscraper with an expansive view ofthe city outside. Objects within the environment include black leathersofas, marble tables Moroccan pillows, flameless flickering candles,mirrored runners, sleek bowls of Albanian dried fruits and florals.Other objects include state of the art electronics across the wallsfeaturing touch screen features like touch appliances, flat screens,refrigerators and sound systems. The environment can also be created ina studio or soundstage. The objects within the environment may bemodified depending on the particular show being recorded. A studio orsoundstage environment may be modified to create a living room, sunroom,bedroom, patio, or other home scene, or other environments that areknown in the television business.

Objects as used in this embodiment refer to products such as the tableand chairs 14, orbs 22, set of chairs 24, and misting umbrella set 28.The exemplary objects depicted in the environment 2 are not intended tobe limiting and any product could be placed in the environment 2. Beyondproducts, objects also include any other physical item that is placed inthe environment, such as a physical advertisement including a banner orlight board. Objects are used to trigger additional content to bedisplayed on the screen as described in more detail herein.

According to the example embodiment, the environment 2 presents a scenethat is used to record video content for a television or streaming showsuch as a talk show. The video camera 32 records the actions of the cast36 as the cast 36 interacts in the environment 2 and, optionally, theobjects therein. The objects are placed within the environment 2 in sucha manner as to be in the line of sight of the video camera 32 so that asthe video camera 32 records the show, the objects appear in the videocontent. The cast is not required to be a part of the scene. B-roll,which is video created to be used in connection with a television show,is one example where the cast would not be included. Alternatively, thevideo camera 32 may be used to record a scene based on the environment 2with objects.

As used herein the video camera 32 can be any range of devices thatrecord video, and preferably audio. Example video cameras include studiocameras, field cameras, camcorders, mobile phones, and tablets. Anexample digital camera that may be used is the DIGI SUPER 8611 fromCanon. The video camera 32 may be digital or analog and may or may notbe connected to a network. In the example embodiment, one video camerais used, but any number of video cameras may be used. Multiple cameraproducts (MCP) generally uses multiple studio cameras that are linked toa control unit.

The video camera 32 creates video content, which includes video andaudio. The video content may be uploaded to a network via networkcapabilities of the video camera 32, using a computer, or using a devicesuch as an adapter that can link to the video camera 32 and connecteither wirelessly or via wire to a network connection.

Referring now to FIG. 2 , an example block diagram of a computing system104 is shown that is useable to implement aspects of the on-demandinteractive content system described herein. In the embodiment shown,the computing system 104 includes at least one central processing unit(“CPU”) 106, a system memory 114, and a system bus 112 that couples thesystem memory 114 to the CPU 106. The system memory 114 includes arandom access memory (“RAM”) 116 and a read-only memory (“ROM”) 118. Abasic input/output system that contains the basic routines that help totransfer information between elements within the computing system 104,such as during startup, is stored in the ROM 118. The computing system104 further includes a mass storage device 120. The mass storage device120 is able to store software instructions and data.

The mass storage device 120 is connected to the CPU 106 through a massstorage controller (not shown) connected to the system bus 112. The massstorage device 120 and its associated computer-readable storage mediaprovide non-volatile, non-transitory data storage for the computingsystem 104. Although the description of computer-readable storage mediacontained herein refers to a mass storage device, such as a hard disk orsolid state disk, it should be appreciated by those skilled in the artthat computer-readable data storage media can include any availabletangible, physical device or article of manufacture from which the CPU106 can read data and/or instructions. In certain embodiments, thecomputer-readable storage media comprises entirely non-transitory media.

Computer-readable storage media include volatile and non-volatile,removable and non-removable media implemented in any method ortechnology for storage of information such as computer-readable softwareinstructions, data structures, program modules or other data. Exampletypes of computer-readable data storage media include, but are notlimited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid statememory technology, CD-ROMs, digital versatile discs (“DVDs”), otheroptical storage media, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed bythe computing system 104.

According to various embodiments of the invention, the computing system104 may operate in a networked environment using logical connections toremote network devices through a network 102, such as a wirelessnetwork, the Internet, or another type of network. The computing system104 may connect to the network 102 through a network interface unit 108connected to the system bus 112. It should be appreciated that thenetwork interface unit 108 may also be utilized to connect to othertypes of networks and remote computing systems. The computing system 104also includes an input/output controller 110 for receiving andprocessing input from a number of other devices, including a touch userinterface display screen, or another type of input device. Similarly,the input/output controller 110 may provide output to a touch userinterface display screen or other type of output device.

As mentioned briefly above, the mass storage device 120 and the RAM 116of the computing system 104 can store software instructions and data.The software instructions include an operating system 124 suitable forcontrolling the operation of the computing system 104. The mass storagedevice 120 and/or the RAM 116 also store software instructions, thatwhen executed by the CPU 106, cause the computing system 104 to providethe functionality discussed in this document. For example, the massstorage device 120 and/or the RAM 116 can store software instructionsthat, when executed by the CPU 106, cause the computing system 104 toreceive and analyze inventory and demand data.

FIG. 3 illustrates a system for generating and providing on-demandinteractive content to an audience, and in particular content related toobjects available for purchase, according to an example embodiment. Thenetwork 102 described in FIG. 2 is connected to a video capture system1002. The video capture system 1002 may be a general purpose computer asset forth in FIG. 2 . In one embodiment, the video data captured fromthe video camera 32 is uploaded to the video capture system 1002. Thevideo capture system 1002 is connected to the network 102. The network102 is connected to a front end system 1004. The front end system may bedirectly access and interacted with by a user to receive or utilize backend capabilities of the system. For example, the front and system 1004may provide an accessible interface at which or may route traffic fromvarious user devices (described below) to an application server 1014,which may access various content from the content server 1006 fordelivery to the respective user devices.

The front end system 1004 can be a software application or thecombination of hardware, software, and network resources. The front endsystem 1004 is connected to a content server 1006, which includes avideo content database 1008, a product information database 1010, and anadvertising information database 1012. The front end 1004 is alsoconnected to an application server 1014.

A video editing computer 1036 is connected to the network 102. The videoediting computer 1036 can have the structure of the computer system 104set forth in FIG. 2 . The video editing computer 1036 interacts with thevideo data stored on the video database 1008, the product data stored inthe product database 1010, and advertising data stored in theadvertising database 1012.

Users may interact with the video data and any content enrichment dataon the on-demand interactive content system set forth in FIG. 3 throughthe use of an interactive device such as a television 1018, a mobilephone 1020, a tablet 1022, a desktop computer 1024, or a laptop computer1026. These devices are exemplary and not intended to be limiting on thedefinition of an interactive device. The interactive devices may displayvideo data 1028, including video data 1028 enhanced with product data1032, advertising data 1034 and other data. Video data 1028 enhancedwith product data 1032 and advertising data 1034 is displayed on theinteractive devices in FIG. 3 . The video content 1028 is enhanced withthe product data 1032 and advertising data 1034 using the video editingcomputer 1036. In the example implementations, the various user devicesdescribed herein may access enriched video content from the applicationserver 1014, which can, in response to requests from the user devices,deliver content from the content of server 1006. The content caninclude, for example, video content from the video content database1008, products content from the products database 1010, andadvertisements from the ads database 1012.

An ecommerce server 1016 is connected to the network 102. The ecommerceserver may be configured as the computer system 104 set forth in FIG. 2. The ecommerce server 1016 operates the ecommerce functions, includingwebsite and purchase options. In example embodiments, the ecommerceserver 1016 may be associated with the entity providing content via thecontent server 1006 and application server 1014. In other embodiments,the ecommerce server 1016 may represent a third party online retailer.

FIG. 4 illustrates the operation of the content server 1006 as set forthin FIG. 3 together with the network 102, ecommerce server 1016, andinteractive devices, including the television 1018, mobile phone 1020,tablet 1022, desktop computer 1024, and laptop computer 1026. FIG. 4illustrates an embodiment of the content server 1006 of FIG. 3 inconnection with the network 102 without optional front end 1004. Thecontent server 1006 includes a processor 102 communicatively connectedto a memory 104 via a data bus 106. The processor 102 can be any of avariety of types of programmable circuits capable of executingcomputer-readable instructions to perform various tasks, such asmathematical and communication tasks.

The memory 104 can include any of a variety of memory devices, such asusing various types of computer-readable or computer storage media. Acomputer storage medium or computer-readable medium may be any mediumthat can contain or store the program for use by or in connection withthe instruction execution system, apparatus, or device. By way ofexample, computer storage media may include dynamic random access memory(DRAM) or variants thereof, solid state memory, read-only memory (ROM),electrically-erasable programmable ROM, optical discs (e.g., CD-ROMs,DVDs, etc.), magnetic disks (e.g., hard disks, floppy disks, etc.),magnetic tapes, and other types of devices and/or articles ofmanufacture that store data. Computer storage media generally includesat least one or more tangible media or devices. Computer storage mediacan, in some embodiments, include embodiments including entirelynon-transitory components.

In the embodiment shown, the memory 104 stores a content developmenttool 112, discussed in further detail below. The content server 1006 canalso include a communication interface 108 configured to receive andtransmit data, for example to access data in an external database, or toprovide product details to interactive devices such as the tablet 1030.Additionally, a display 110 can be used for viewing product detailsgenerated by the content development tool 112.

In various embodiments, the content development tool 112 performs aprocess including enriching video data 1028 using content enrichmentdata 130 such as advertising data 1032, product data 1030 and audio data1048.

In one sample embodiment, the content development tool includes an API114, a video processing subsystem 116, customer API 118, geo location120, tag generator 122, and video overlay 124.

The API 114 provides an access interface at which content may bereceived, for processing by the content development tool 112. The videoprocessing subsystem 116 can receive the video content and process thevideo content to identify one or more objects within the video content.The objects may be, for example, any object that appears in one or morescenes of the video content and may be objects of interest or topics ofdiscussion by the cast. In some instances, the objects may be productsthat have corresponding product listings included in the product data1030. Accordingly, metadata may be applied by the video processingsubsystem 116 to identify the associated objects and products, and togenerate content enrichment data 130 which may define actions to betaken when the video content is played back that may enrich that videocontent. Such actions may include, for example, displaying an overlay ofproduct information based on information in the product data 1030associated with the product that correlates to the object.

In some embodiments, a customer API 118 may allow a user device toaccess content. The customer API 118 may receive requests, for examplefrom an application server 1014 or directly from a user device, and mayrespond with video content for playback as well as one or more contentenrichment actions to be executed in combination with playback of thevideo content.

The geo location system 120 may be used to identify a specific locationthat is displayed in a scene, or alternatively, a specific location of auser device that is used to request playback. In the embodiments inwhich the geo location system 120 identifies a specific location of anobject in the scene, the system may include one or more beacons used toidentify relative positioning of objects and individuals within aconfined geographic location. Accordingly, the geo location system 120may be used to further apply metadata to the video content.

The tag generator 122 generates one or more tags that are displayable incombination with the video content. In example embodiments, the videoprocessing subsystem 116 may identify one or more objects in videocontent, and can identify products that may be associated with thoseobjects. The tag generator 122 can in turn define supplemental contentthat may be displayable in combination with the video content. Thedisplay may include a pop-up overlay providing supplemental content(e.g., text or image content) that can be displayed in proximity to theobject, and may include supplemental information about the product thatis associated with the object. The video overlay 124 is then responsiveto access via the customer API 118 from user devices, and can serve thevideo content and any related content enrichment data, including tagcontent, for display alongside or over the video content when requestedby a user device.

In one embodiment, as described in more detail below, the video camera32 may be used to take close ups of each object in the scene prior to orafter recording the content. The close ups of the objects may be used toimage match the objects to the product data, advertising data, audiodata, and other content enrichment data. After the video data 1028 iscollected using the video camera, the computer system (e.g., the videoprocessing subsystem 116) uses image recognition software to match eachobject that was subject to a close up with the object as shown in thevideo data.

In the embodiment shown, the content development tool 112 receives acombination of the advertising data 1048 from the advertising database1012, the product data 1030 from the product database 1010, and thevideo data 1028 from the video database 1008. Content enrichment data130 may be stored in any of the video database 1008, product database1010, or advertising database 1012, or separately from those databases.While this embodiment separates out data into different databases, oneof ordinary skill would understand that the data could be includes in adifferent combination of databases.

The video data 1028 is generated from the video camera 32 set forth inFIG. 1 . In one sample embodiment, the video camera 32 is fitted with areceiver that receives signals from a beacon. Beacons may be placed onobjects within the scene, such as the chairs 24. As the video camera 32pans towards the chairs 24, the beacon on the chairs activates the videocamera 32 to tag the video with a tag that identifies the specificbeacon on the chairs. That beacon can be linked to particular productdata, advertising data, and audio data.

The network 102 may also include an ecommerce server 1016 or communicatewith an ecommerce server 1016 remote from the network 102. The blockdiagram of a computing system 104 may be used to implement aspects ofthe ecommerce server described herein. The ecommerce server 1016comprises a system configures to facilitate the buying or selling ofproducts over a network. The ecommerce server 1016 can be local to orincluded in the network 100 such as the extranet of an ecommerce entity.

FIG. 5 illustrates an example of the breakdown of video content that canbe obtained via video camera 32. A video 1038 can be broken down intoscenes 1040. Scenes 1040 can be broken down into shots 1042. Shots canbe broken down into frames 1044. Each frame may be edited individuallyor as a group. FIG. 5 is exemplary and is not intended to be to scale ofthe number of frames that are in a single shot.

In the example shown, it is noted that objects may appear in one or moreframes, shots, or scenes. For example, a particular object may appearduring a portion of two scenes, in a subset of shots taken during thosescenes, and in a specified set of frames. Video editing software mayautomatically identify that object, match the object to a particularproduct, and, upon confirmation by a user, display specified taginformation (e.g., product information) either during the display of theobject, during the scene or shot in which the object is displayed, orthrough some other portion (or all) of the video 1038. Such timing maybe defined in the content enrichment data 130 described above.

FIG. 6 illustrates an example of video editing software 1046 that can beused to edit video data 1028. In the example shown, product data 1030and advertising data 1032 may be associated with the video data 1028 forconcurrent display during playback of the video data on one or more userdevices. Content enrichment data 130 can be edited into the video data1028 such as product data 1032, advertising data audio data 1048. Thevideo editing software 1046 displayed in FIG. 6 can be used to edit theframes 1044 of the video data 1028 in a manner to add content enrichmentdata 130. In the embodiment shown, product data 1032 in the form of aproduct tag is added to the video data 1028. Audio data 1048 is added tothe video data 1028. Advertising data 1032 is added to the video data1028 in the form of an advertisement that appears in the lower leftcorner of the display. One or more frames may be edited in such a mannerto include the content enrichment data on the screen for a specifiedamount of time. Addition of such content enrichment data can be done inan at least partially automated manner, e.g., by automaticallyidentifying objects in the video data, and correlating those objects toproducts and or advertisements previously stored in the product andadvertisement databases 1010, 1012 respectively.

FIGS. 7-10 illustrate the addition of content enrichment data 130 tovideo data 1028. Each of FIGS. 7-10 represents a single frame of thevideo data 1028. FIG. 7 represents a first frame of video data 1028wherein there are no objects. FIG. 8 is a second frame that includes thecast 36 walking toward chairs 24. In the embodiment, the chairs 24 aredisplayed together with product data in the form of a tag on the screen.The chairs 24 are also displayed with the advertisement 1032 related tothe chairs 24.

FIG. 9 represents a third frame that includes the cast 36 continuing towalk in the scene past the chairs 24. As the cast 36 walks past thechairs 24, the misting umbrella 28 comes into view. As the mistingumbrella 28 comes into view, product data 1031 related to the mistingumbrella is displayed on the screen together with the misting umbrella1031. Advertising data 1033 is also displayed relative to the object inthe screen, in this case the misting umbrella 28.

FIG. 10 illustrates the hand 1050 of a user of the interactive deviceselecting the misting umbrella 28 for purchase. Upon selection, the usermay automatically purchase the misting umbrella 28. The user may alsomodify the settings of the system to allow the user to be transferred toa website or other ecommerce platform to learn additional informationabout the product and to make the purchase of the product from thewebsite or other ecommerce platform. Other options to initiate apurchase of an object include using a purchase code that is provided onscreen, use an app to purchase the object that is live on the screen, orbring up a new screen on the interactive device to make a purchase,which may optionally automatically pause the video playback. Theseexamples are not meant to be exhaustive and one can understand there maybe other methods to initiate the purchase of a product. Users also mayuse various forms to engage a purchase, including clicking, swiping,touching, using audible sounds, or using a separate device to engage thepurchase.

While the embodiment uses certain objects such as chairs 24 and themisting umbrella 28 to illustrate objects that may be purchased, anynumber of objects may be subject to purchase. These may include fooditems that are shown on screen and upon selection may engage a deliveryservice to deliver the items on screen. Selection of a food item mayalso provide product data related to recipes associated with the food,which may be delivered to the user upon selection of the food item.

It should be understood that the playback of the enhanced video data canbe transmitted on a tape delay, streaming service, live or any otherplayback system. Additionally, the enhanced video data may include videodata and associated content enrichment data, or may be enriched videodata with such enrichment (e.g., tags, advertisements, etc.) built intothe video data stream.

It should also be understood that a user may interact with the enhancedvideo data, including moving the enhanced video data forward, reverse,zoom in, zoom out, and pause. Another option is to active all productdata for objects on the screen when the pause button is activated.

FIG. 11 illustrates one embodiment wherein the objects in theenvironment are recognized using photo capture technology. A videocamera 32 or still shot camera 33 may be used to capture images of theobjects in an environment. By way of example, objects in FIG. 11 includea vase 38 on a shelf 40, a frame 42, a television 44, a mantle 46, afireplace 48, a book shelf 50, a bean bag 52, a couch 54, and a pillow56. FIG. 3 shows the option of using a video camera 32 or a still shotcamera 33 to take still video or photographs of the objects. While twoexamples of a video camera 32 and still shot camera 33 are used, anydevice capable of capturing a photo, frame of video, or visualrepresentation of an object may be used. In one embodiment, the stillshot camera 33 takes a photo of each of the objects. In FIG. 3 , a photoof the vase 38 on a shelf 40 is depicted in a vase photo 58. A photo ofthe frame 60 is depicted in a frame photo 60. A photo of the television44 is depicted in a television photo 62. A photo of the book shelf 50 isdepicted in a book shelf photo 64. A photo of the beanbag 52 is depictedin a beanbag photo 66. A photo of the fireplace 48 is depicted in thefireplace photo 68. A photo of the pillow 56 is depicted in the pillowphoto 70. Any number of objects can be placed in the environment and maybe the subject of photographs. The photographs 72 may be stored in theproduct database 1010 as shown in FIG. 3 . Additionally, available imagerecognition technologies may be utilized to correlate the objects in thephotographs to the same or similar objects appearing in the video data,thereby allowing for automatic recognition of the objects withinportions of the video data, on a frame by frame basis.

FIG. 12 illustrates the identification of objects within video data 1028using the photographs 72 described in FIG. 11 . The photographs 72 arestored in the product database 1010. A computer system 104 as set forthin FIG. 2 may be used to process the video data 1028 stored in the videodatabase 1008 and identify the frames in which each object subject tothe photographs 72 appears. The computer system will identify thespecific location in each frame where the object appears based on imageidentification technology. That location in the video is recorded in theproduct database 1010. Other product information related to the objectis also located in the product database 1010 including a productheading, description of the product, and product cost. Other informationmay also be included (e.g., in a tag or other overlay of information onthe video data as included in the content enrichment data).

FIG. 13 illustrates another embodiment using a grid layout. The gridlayout is separated into horizontal and vertical grids. An object withinthe grid is set forth in one or more portions of the grid pattern. Anobject may overlap a single portion of the grid. The object locationaccording to the grid is stored in the product database 1010 and is usedto associate product data with a particular object within a particularlocation within a frame.

In the ecommerce solutions described herein, the object that appearsonline may be available for purchase. There are scenarios where aproduct is sold out or no longer available. In those circumstances,replacement products may be made available. One embodiment is toidentify alternative or replacement options in the product data. When aproduct is sold out, the replacement product is made available in itsplace. This may be used in connection with on screen purchases as welltogether with an optional note.

FIG. 14 shows a general flowchart of a method 200 of receiving andprocessing program content for enrichment and playback, in accordancewith example aspects of the present disclosure. In the embodiment shown,the method 200 includes receiving captured video content (step 202).Receiving captured video content can include, for example, video contentdisplaying a plurality of scenes, the plurality of scenes displaying acast and one or more objects included in one or more of the scenes.

In the embodiment shown, the method 200 includes ingesting the videocontent into a content database (step 204). Once in the contentdatabase, the video content may be processed to automatically identifyone or more objects that are displayed in one or more scenes of thevideo content (step 206). This can include, for example, use of one ormore commercially available object recognition technologies, such asVision AI, from Google. Other object recognition technologies may beused as well. Once objects are recognized, those objects may beautomatically correlated to one or more products that are known to thecontent of server. The products can be, for example, products havingproduct information stored in a product database, or having productinformation stored in an e-commerce server accessible from the contentserver.

In the embodiment shown, the method 200 further includes defining one ormore content enrichment actions that may be performed during playback ofthe content when the content is requested and playback by a user device(step 208). The one or more content enrichment actions can include, forexample, generation of an overlay that provides product details of theproduct that is associated with the object. In other examples, the oneor more content enrichment actions can include display of one or moreadvertisements associated with the object or a product related to theobject.

In example embodiments, the overlay may be defined as a tag which isdisplayed over the video content in the vicinity of the object while theobject appears in the video content during playback. In other examples,the overlay may be text (e.g., subtitles or other overlay text) whichotherwise provides supplemental information to a viewer who may wish tolearn more about a topic of conversation occurring in the video contentby or among the cast. Such content enrichment actions may be defined bya user having video editing tools, as discussed above. The user may beprompted to define such content enrichment actions upon the contentdevelopment tool 112 automatically identifying objects having matchedproducts or advertisements.

FIG. 15 shows a general flowchart of a method 300 of interacting withenriched video content in accordance with the methods and systemsdescribed herein. The method 300 may be initiated by an interactive userdevice, such as any of the devices previously described in conjunctionwith FIG. 3 .

In the embodiment shown, the method 300 includes receiving a requestfrom an interactive user device for playback of video content (step302). The request may be received at an application server and forwardedto a content server (e.g., for receipt by a customer API 118 of contentdevelopment tool 112). In response to the request, the method 300 mayfurther include obtaining the requested video content, alongside relatedproduct information, and optionally advertisement information (step304). A content feed may be generated and provided to an interactiveuser device that includes not only the original video content, butadditionally the content enrichment actions, to provide enriched videocontent (step 306). This can include, for example, displaying an optionto purchase a product corresponding to an object while that object isdisplayed during playback of the video content. It can also includedisplaying an advertisement associated with an object while the objectis displayed during playback. Still further, displaying enriched videocontent may include displaying an advertisement associated with a topicof conversation while that topic of conversation is addressed during theprogramming reflected in the video content. For example, the topic ofconversation may be a topic addressed during a talk show. Theadvertisement, or product information, may be displayed as an overlay,for example a tag, displayed over a portion of the video content.

Still further, in the method 300, one or more interactions may bereceived from a in the interactive user device (step 308). Theinteractions may include, for example, a selection of a tag or overlayor advertisement relating to a particular product that was previouslyautomatically identified as being related to an object appearing in thevideo content. In accordance with the present disclosure, interactionsmay be provided by a user in various forms, for example using touch,gesture, voice activation, cursor, or other types of selections forobjects within a graphical interface displaying an audiovisualpresentation (e.g., of the enriched video content).

In response to the received interactions, the method of 300 may furtherinclude automatically generating one or more product related actionsbased on the interaction (step 310). The product related interactionsmay include, for example, automatically generating and forwarding arequest to purchase a product associated with the object to ane-commerce server. The product related interactions may also includedisplaying supplemental information about a product in response to userselection of the product, display of one or more advertisements orpromotions associated with a product, or other similar actions.

Referring now to FIGS. 1-15 generally, it is noted that the presentapplication has a number of advantages over existing television-based,video-based, or static online retail systems. For example, as comparedto existing static product display web sites and television basedshopping experiences, the present application allows user interactionwith video content to allow the user to discover products through thatuser's own curiosity without making the object or product oradvertisement the focal point of the video content. Accordingly, userswill be more likely to investigate products on their own since the videocontent will address the topic of interest and the product informationcan be presented in a way that is subsidiary to the topic of discussionoccurring during the video content. Therefore, the user's interest isretained for a longer period of time, allowing more objects, andtherefore more products and advertisements, to be displayed to the user.

Embodiments of the present invention, for example, are described abovewith reference to block diagrams and/or operational illustrations ofmethods, systems, and computer program products according to embodimentsof the invention. The functions/acts noted in the blocks may occur outof the order as shown in any flowchart. For example, two blocks shown insuccession may in fact be executed substantially concurrently or theblocks may sometimes be executed in the reverse order, depending uponthe functionality/acts involved.

The description and illustration of one or more embodiments provided inthis application are not intended to limit or restrict the scope of theinvention as claimed in any way. The embodiments, examples, and detailsprovided in this application are considered sufficient to conveypossession and enable others to make and use the best mode of claimedinvention. The claimed invention should not be construed as beinglimited to any embodiment, example, or detail provided in thisapplication. Regardless of whether shown and described in combination orseparately, the various features (both structural and methodological)are intended to be selectively included or omitted to produce anembodiment with a particular set of features.

Having been provided with the description and illustration of thepresent application, one skilled in the art may envision variations,modifications, and alternate embodiments falling within the spirit ofthe broader aspects of the claimed invention and the general inventiveconcept embodied in this application that do not depart from the broaderscope.

1. A system for generating and providing on-demand interactive contentto an audience, the system comprising: a content server including aprocessor and a memory, the memory hosting a content database, a productdatabase, and an advertising database, the memory further storinginstructions which, when executed, cause the content server to: receivevideo content, the video content including a talk show having aplurality of scenes, each of the scenes including a host, one or moreguests, and a plurality of objects; ingest the video content into thecontent database; process the video content to automatically identify anobject of the plurality of objects as associated with a product havingan associated product record in the product database; define one or morecontent enrichment actions to be performed during playback of thecontent, the one or more content enrichment actions including generatingan overlay providing product details of the product based on informationabout the product in the product database, the overlay being displayedin association with the content during playback of a portion of thecontent during which the object appears; an application servercommunicatively connected to the content server and configured toexecute instructions which cause the application server to perform:receive a request from an interactive user device for playback of thevideo content; in response to the request, obtaining content, productinformation, and advertisement information from the content server; andgenerate a content feed to the interactive user device, wherein thecontent feed allows interaction with the content feed during playback ofthe video content based on the one or more content enrichment actions.2. The system of claim 1, wherein at least one content enrichment actionincludes presenting to a user an option to purchase the productassociated with the object from a retailer.
 3. The system of claim 2,wherein the retailer comprises an online retailer.
 4. The system ofclaim 1, wherein the at least one content enrichment action includespresenting to the user a link to the product associated with the object,the link being to an online retail website.
 5. The system of claim 1,wherein the plurality of objects each is associated with at least one ofa product and an advertisement.
 6. The system of claim 1, wherein theproduct details are included in product information and the overlaycomprises a tag displayed in association with the object, the overlaybeing defined within content enrichment data that is generated duringprocessing of the video content and which defines the one or morecontent enrichment actions.
 7. The system of claim 1, wherein thecontent server is further configured to define one or moreadvertisements to be displayed in association with the content duringplayback of at least a portion of the content.
 8. The system of claim 1,further comprising at least one interactive user device communicativelyconnected to the content server and the application server, the at leastone interactive user device having software thereon capable of playbackof the content feed.
 9. The system of claim 8, wherein the content feedcomprises enriched video content.
 10. The system of claim 8, wherein theat least one interactive user device is selected from the group ofdevices consisting of: a television, a set-top box, a phone, a tablet,and a portable computing system.
 11. The system of claim 8, wherein theat least one interactive user device comprises a touch screen device,and wherein, in response to user selection of a tag associated with anobject displayed during playback of the video content and displayed asone of the one or more content enrichment actions, the user is presentedwith an option to purchase the product associated with the object. 12.The system of claim 11, wherein, in response to user selection of thetag, an automated purchase request is transmitted from at least one ofthe interactive user device, the application server, and the contentserver to an e-commerce server.
 13. A method of creating interactiveretail-focused video content, the method comprising: capturing videocontent of a talk show having a plurality of scenes, each of the scenesincluding a host, one or more guests, and a plurality of objects;ingesting the video content into a content database; processing thevideo content to automatically identify an object of the plurality ofobjects as associated with a product having an associated product recordin a product database; defining one or more content enrichment actionsto be performed during playback of the content, the one or more contentenrichment actions including generating an overlay providing productdetails of the product based on information about the product in theproduct database, the overlay being displayed in association with thecontent during playback of a portion of the content during which theobject appears; defining an advertisement to be displayed in associationwith the content during playback of at least a portion of the content;upon receipt of a request for the video content, providing the videocontent, the one or more content enrichment actions, and theadvertisement for playback.
 14. The method of claim 13, wherein theadvertisement is associated with a second object of the plurality ofobjects, and is displayed during playback of a portion of the contentduring which the second object is displayed.
 15. A method of providinginteractive retail-focused video content for playback, the methodcomprising: receiving, from an application executing on a consumercomputing device, a request for interactive video content, the requestidentifying specified video content, the specified video contentcomprising a talk show having a plurality of scenes, each of the scenesincluding a host, one or more guests, and a plurality of objects;retrieving the specified video content as well as product data,advertising data, and one or more enrichment actions to be performedduring playback of the specified video content; providing interactivevideo content to the consumer computing device, the interactive videocontent comprising the specified video content and executing the one ormore enrichment actions during playback of the content, the one or moreenrichment actions including one or more of: displaying an option topurchase a product corresponding to an object of the plurality ofobjects while the object is displayed during playback of the specifiedvideo content; displaying an advertisement associated with an object ofthe plurality of objects while the object is displayed during playbackof the specified video content; and displaying an advertisementassociated with a topic of conversation while the topic of conversationis addressed during the talk show as an overlay on a portion of thespecified video content.